One Possible Solution

To optimize for few false positives (misclassified, valid transactions), I defined a model that accounts for class imbalance and optimizes for a high precision .

Let's review the scenario:

A bank has asked you to build a model that optimizes for a good user experience; users should only ever have up to about 15% of their valid transactions flagged as fraudulent.

My thoughts : If we're allowed about 15/100 incorrectly classified valid transactions (false positives), then I can calculate an approximate value for the precision that I want as: 85/(85+15) = 85%. I'll aim for about 5% higher during training to ensure that I get closer to 80-85% precision on the test data.

%%time
# One possible solution
# instantiate and train a LinearLearner

# include params for tuning for higher precision
# *and* account for class imbalance in training data
linear_precision = LinearLearner(role=role,
                                train_instance_count=1, 
                                train_instance_type='ml.c4.xlarge',
                                predictor_type='binary_classifier',
                                output_path=output_path,
                                sagemaker_session=sagemaker_session,
                                epochs=15,
                                binary_classifier_model_selection_criteria='recall_at_target_precision',
                                target_precision=0.9,
                                positive_example_weight_mult='balanced')


# train the estimator on formatted training data
linear_precision.fit(formatted_train_data) 

This model trains for a fixed precision of 90%, and, under that constraint, tries to get as high a recall value as possible. After training, I deployed the model to create a predictor:

%%time 
# deploy and evaluate a predictor
precision_predictor = linear_precision.deploy(initial_instance_count=1, instance_type='ml.t2.medium')

INFO:sagemaker:Creating model with name: linear-learner-2019-03-11-04-07-10-993
INFO:sagemaker:Creating endpoint with name linear-learner-2019-03-11-03-36-56-524

Then evaluated the model by seeing how it performed on test data:

print('Metrics for tuned (precision), LinearLearner.\n')

# get metrics for balanced predictor
metrics = evaluate(precision_predictor, 
                   test_features.astype('float32'), 
                   test_labels, 
                   verbose=True)

These were the results I got:

Metrics for tuned (precision), LinearLearner.

prediction (col)    0.0  1.0
actual (row)                
0.0               85276   26
1.0                  31  110

Recall:     0.780
Precision:  0.809
Accuracy:   0.999

As you can see, we still misclassified 26 of the valid results and so I may have to go back and up my aimed-for precision; the recall and accuracy are not too bad, considering the precision tradeoff.

Finally, I made sure to delete the endpoint after I was doe with evaluation.

## IMPORTANT
# delete the predictor endpoint 
delete_endpoint(precision_predictor)

Deleted linear-learner-2019-03-11-03-36-56-524